3D Pointer Mapping

ABSTRACT

Systems, devices, methods and software are described for mapping movement or motion of a 3D pointing device into cursor position, e.g., for use in rendering the cursor on a display. Absolute and relative type mapping algorithms are described. Mapping algorithms can be combined to obtain beneficial characteristics from different types of mapping.

RELATED APPLICATION

This application is related to, and claims priority from, U.S.Provisional Patent Application Ser. No. 61/077,238, entitled “3D PointerMapping”, filed on Jul. 1, 2008, the disclosure of which is incorporatedhere by reference.

BACKGROUND

The present invention describes mapping techniques, systems, softwareand devices, which can be used in 3D pointing devices, as well as inother types of devices.

Technologies associated with the communication of information haveevolved rapidly over the last several decades. Television, cellulartelephony, the Internet and optical communication techniques (to namejust a few things) combine to inundate consumers with availableinformation and entertainment options. Taking television as an example,the last three decades have seen the introduction of cable televisionservice, satellite television service, pay-per-view movies andvideo-on-demand. Whereas television viewers of the 1960s could typicallyreceive perhaps four or five over-the-air TV channels on theirtelevision sets, today's TV watchers have the opportunity to select fromhundreds, thousands, and potentially millions of channels of shows andinformation. Video-on-demand technology, currently used primarily inhotels and the like, provides the potential for in-home entertainmentselection from among thousands of movie titles.

The technological ability to provide so much information and content toend users provides both opportunities and challenges to system designersand service providers. One challenge is that while end users typicallyprefer having more choices rather than fewer, this preference iscounterweighted by their desire that the selection process be both fastand simple. Unfortunately, the development of the systems and interfacesby which end users access media items has resulted in selectionprocesses which are neither fast nor simple. Consider again the exampleof television programs. When television was in its infancy, determiningwhich program to watch was a relatively simple process primarily due tothe small number of choices. One would consult a printed guide which wasformatted, for example, as series of columns and rows which showed thecorrespondence between (1) nearby television channels, (2) programsbeing transmitted on those channels and (3) date and time. Thetelevision was tuned to the desired channel by adjusting a tuner knoband the viewer watched the selected program. Later, remote controldevices were introduced that permitted viewers to tune the televisionfrom a distance. This addition to the user-television interface createdthe phenomenon known as “channel surfing” whereby a viewer could rapidlyview short segments being broadcast on a number of channels to quicklylearn what programs were available at any given time.

Despite the fact that the number of channels and amount of viewablecontent has dramatically increased, the generally available userinterface, control device options and frameworks for televisions has notchanged much over the last 30 years. Printed guides are still the mostprevalent mechanism for conveying programming information. The multiplebutton remote control with up and down arrows is still the mostprevalent channel/content selection mechanism. The reaction of those whodesign and implement the TV user interface to the increase in availablemedia content has been a straightforward extension of the existingselection procedures and interface objects. Thus, the number of rows inthe printed guides has been increased to accommodate more channels. Thenumber of buttons on the remote control devices has been increased tosupport additional functionality and content handling, e.g., as shown inFIG. 1. However, this approach has significantly increased both the timerequired for a viewer to review the available information and thecomplexity of actions required to implement a selection. Arguably, thecumbersome nature of the existing interface has hampered commercialimplementation of some services, e.g., video-on-demand, since consumersare resistant to new services that will add complexity to an interfacethat they view as already too slow and complex.

In addition to increases in bandwidth and content, the user interfacebottleneck problem is being exacerbated by the aggregation oftechnologies. Consumers are reacting positively to having the option ofbuying integrated systems rather than a number of separate components.An example of this trend is the combination television/VCR/DVD in whichthree previously independent components are frequently sold today as anintegrated unit. This trend is likely to continue with an end resultthat potentially all of the communication devices currently found in thehousehold will be packaged together as an integrated unit, e.g., atelevision/VCR/DVD/internet access/radio/stereo unit. Even those whocontinue to buy separate components will likely desire seamless controlof, and interworking between, the separate components. With thisincreased aggregation comes the potential for more complexity in theuser interface. For example, when so-called “universal” remote unitswere introduced, e.g., to combine the functionality of TV remote unitsand VCR remote units, the number of buttons on these universal remoteunits was typically more than the number of buttons on either the TVremote unit or VCR remote unit individually. This added number ofbuttons and functionality makes it very difficult to control anythingbut the simplest aspects of a TV or VCR without hunting for exactly theright button on the remote. Many times, these universal remotes do notprovide enough buttons to access many levels of control or featuresunique to certain TVs. In these cases, the original device remote unitis still needed, and the original hassle of handling multiple remotesremains due to user interface issues arising from the complexity ofaggregation. Some remote units have addressed this problem by adding“soft” buttons that can be programmed with the expert commands. Thesesoft buttons sometimes have accompanying LCD displays to indicate theiraction. These too have the flaw that they are difficult to use withoutlooking away from the TV to the remote control. Yet another flaw inthese remote units is the use of modes in an attempt to reduce thenumber of buttons. In these “moded” universal remote units, a specialbutton exists to select whether the remote should communicate with theTV, DVD player, cable set-top box, VCR, etc. This causes many usabilityissues including sending commands to the wrong device, forcing the userto look at the remote to make sure that it is in the right mode, and itdoes not provide any simplification to the integration of multipledevices. The most advanced of these universal remote units provide someintegration by allowing the user to program sequences of commands tomultiple devices into the remote. This is such a difficult task thatmany users hire professional installers to program their universalremote units.

Some attempts have also been made to modernize the display interfacebetween end users and media systems. However, these attempts typicallysuffer from, among other drawbacks, an inability to easily scale betweenlarge collections of media items and small collections of media items.For example, interfaces which rely on lists of items may work well forsmall collections of media items, but are tedious to browse for largecollections of media items. Interfaces which rely on hierarchicalnavigation (e.g., tree structures) may be speedier to traverse than listinterfaces for large collections of media items, but are not readilyadaptable to small collections of media items. Additionally, users tendto lose interest in selection processes wherein the user has to movethrough three or more layers in a tree structure. For all of thesecases, current remote units make this selection processor even moretedious by forcing the user to repeatedly depress the up and downbuttons to navigate the list or hierarchies. When selection skippingcontrols are available such as page up and page down, the user usuallyhas to look at the remote to find these special buttons or be trained toknow that they even exist. Accordingly, organizing frameworks,techniques and systems which simplify the control and display interfacebetween users and media systems as well as accelerate the selectionprocess, while at the same time permitting service providers to takeadvantage of the increases in available bandwidth to end user equipmentby facilitating the supply of a large number of media items and newservices to the user have been proposed in U.S. patent application Ser.No. 10/768,432, filed on Jan. 30, 2004, entitled “A Control Frameworkwith a Zoomable Graphical User Interface for Organizing, Selecting andLaunching Media Items”, the disclosure of which is incorporated here byreference.

Of particular interest for this specification are the remote devicesusable to interact with such frameworks, as well as other applicationsand systems. As mentioned in the above-incorporated application, variousdifferent types of remote devices can be used with such frameworksincluding, for example, trackballs, “mouse”-type pointing devices, lightpens, etc. However, another category of remote devices which can be usedwith such frameworks (and other applications) is 3D pointing devices.The phrase “3D pointing” is used in this specification to refer to theability of an input device to measure motion in three dimensional space.Three dimensional space has six degrees of freedom (6DOF): three axes oflinear motion and three axes of angular motion. Although the term 6DOFis commonly used, the seventh dimension of time is automaticallyincluded. The position (or pose) of a device may be represented by itslinear position and angular position (orientation). The 3D pointingdevice moves within the six degrees of freedom in the air in front of,e.g., a display, and the corresponding ability of the user interface totranslate those motions directly into user interface commands, e.g.,movement of a cursor on the display. Thus “3D pointing” differs from,e.g., conventional computer mouse pointing techniques which use asurface, e.g., a desk surface or mousepad, as a proxy surface from whichrelative movement of the mouse is translated into cursor movement on thecomputer display. An example of a 3D pointing device can be found inU.S. Pat. No. 7,118,518 to Matthew G. Liberty (hereafter referred to asthe '518 patent), the disclosure of which is incorporated here byreference. The motion data is transferred between the 3D pointing deviceand the host system. The motion data transfer may be performed by anycommunication link including wired, radio frequency, ultrasonic andinfrared.

Two primary methods exist for mapping device motion into cursor motion:absolute pointing and relative pointing. With absolute pointing, thedesired cursor location is the location where the forward vector of thedevice intersects the plane of the display. If the 3D pointer were alaser pointer, this cursor location would be the location of theprojected laser dot. With absolute pointing, the pointing resolution forangular motions varies with linear position. The further the user isfrom the display, the finer the angular motion required to targetobjects. The angular motion resolution also varies with the off-axisangle. When the user is to the side of the display, smaller angularmotions are required than when the user is at the same distance directlyin front of the display. This variation in pointing resolution yields aninconsistent user experience. When part of a TV remote control, thisvariation causes inconsistent behavior between homes and even betweenseats at the same home. Absolute pointing is, however, normallyrepeatable and time invariant. If the 3D pointer is placed in the sameposition, then the cursor will return to the same position. Absolutepointing may also be non-calibrated and referenced to an initialstarting position.

With relative pointing, the pointing resolution for all motion isindependent of linear position and angular position from the display.However, the device may not be aligned with the cursor on the display.Relative pointing allows for non-linear processing including pointerballistics which can dramatically improve pointing performance. Pointerballistics are described, for example, athttp://www.microsoft.com/whdc/archive/pointer-bal.mspx. Relativepointing often bounds cursor motion to the display bounds and discardsany motion beyond the display bounds. While this allows users to relaxand find a comfortable position, some applications benefit from a fixedmapping between device position and cursor location.

However, solutions exist beyond pure absolute pointing and pure relativepointing. Accordingly, there is still room for improvement in the areaof mapping of, e.g., device movement to display, handheld device design,generally, and 3D pointer design, more specifically. For the remainderof this specification, absolute pointing refers to solutions that havecharacteristics most similar to true absolute pointing and relativepointing refers to solutions that have characteristics most similar totrue relative pointing.

SUMMARY

According to one exemplary embodiment, a method for mapping a device'smovement into cursor position is described. The device's linear positionand angular position are estimated. The estimated linear position andangular position are further processed using both a first mappingalgorithm to generate a first cursor location, and a second mappingalgorithm to generate a second cursor location. The first cursorlocation and the second cursor location are combined to generate a finalcursor output. Such a technique can be used, for example, to combine thestrengths of the two mapping algorithms to provide a more robust userexperience associated with, e.g., a user interface in which the cursoris used to interact with various objects.

According to another exemplary embodiment, a 3D pointing device includesat least one sensor configured to generate an output which is associatedwith movement of the 3D pointing device and a processor. The processoris configured to estimate the device's linear position and angularposition using the output, to process the estimated linear position andangular position using both a first mapping algorithm to generate afirst cursor location and a second mapping algorithm to generate asecond cursor location, and to combine the first cursor location and thesecond cursor location to generate a final cursor output.

According to another exemplary embodiment, a system includes a 3Dpointing device having at least one sensor configured to generate anoutput which is associated with movement of said 3D pointing device anda system controller in communication with the 3D pointing device andconfigured to receive data associated with said output therefrom. Atleast one of the 3D pointing device and the system controller include aprocessor for estimating at least one of said device's linear positionand angular position using the output, to process at least one of theestimated linear position and angular position using both a firstmapping algorithm to generate a first cursor location and a secondmapping algorithm to generate a second cursor location, and to combinethe first cursor location and the second cursor location to generate afinal cursor output.

According to another exemplary embodiment, a method for mapping adevice's movement into cursor position includes estimating the device'slinear position and angular position, and processing the estimatedlinear position and angular position using a mapping algorithm togenerate a cursor location, wherein the mapping algorithm is an absoluteinvariant algorithm which has a first characteristic of providing adirect, repeatable mapping from device linear position and angularposition into cursor location and a second characteristic that cursorresponsiveness to linear motion and angular motion is consistent over amotion range.

According to another exemplary embodiment, a method for mapping adevice's movement into cursor position includes the steps of estimatingat least one of the device's linear position and angular position, andprocessing at least one of the estimated linear position and angularposition using a mapping algorithm to generate a cursor location,wherein the mapping algorithm creates an intermediate, virtual displaythat moves to face the device.

According to another exemplary embodiment, a method for mapping adevice's movement into cursor position, includes the steps of estimatingthe device's angular position, and processing the estimated angularposition using a mapping algorithm to generate a cursor location,wherein the mapping algorithm maps said estimated angular position ofthe device into cursor coordinates using an angular position sphericalprojection.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate exemplary embodiments, wherein:

FIG. 1 depicts a conventional remote control unit for an entertainmentsystem;

FIG. 2 depicts an exemplary media system in which exemplary embodimentscan be implemented;

FIG. 3 shows a 3D pointing device according to an exemplary embodimentof the present invention;

FIG. 4 illustrates a cutaway view of the 3D pointing device in FIG. 4including angular velocity sensing and linear acceleration sensing;

FIG. 5 shows a 3D pointing device according to another exemplaryembodiment;

FIG. 6 depicts the 3D pointing device of FIG. 5 being used as part of a“10 foot” interface according to an exemplary embodiment;

FIG. 7 shows mapping of device motion into displayed cursor motionaccording to various exemplary embodiments;

FIGS. 8-12 illustrate functions associated with mapping of device motioninto displayed cursor motion according to exemplary embodiments; and

FIG. 13 is a flowchart illustrating a method for mapping a device'smovement into cursor position according to an exemplary embodiment.

DETAILED DESCRIPTION

The following detailed description of the invention refers to theaccompanying drawings. The same reference numbers in different drawingsidentify the same or similar elements. Also, the following detaileddescription does not limit the invention. Instead, the scope of theinvention is defined by the appended claims.

In order to provide some context for this discussion, an exemplaryaggregated media system 200 in which the present invention can beimplemented will first be described with respect to FIG. 2. Thoseskilled in the art will appreciate, however, that the present inventionis not restricted to implementation in this type of media system andthat more or fewer components can be included therein. Therein, aninput/output (I/O) bus 210 connects the system components in the mediasystem 200 together. The I/O bus 210 represents any of a number ofdifferent of mechanisms and techniques for routing signals between themedia system components. For example, the I/O bus 210 may include anappropriate number of independent audio “patch” cables that route audiosignals, coaxial cables that route video signals, two-wire serial linesor infrared or radio frequency transceivers that route control signals,optical fiber or any other routing mechanisms that route other types ofsignals.

In this exemplary embodiment, the media system 200 includes atelevision/monitor 212, a video cassette recorder (VCR) 214, digitalvideo disk (DVD) recorder/playback device 216, audio/video tuner 218 andcompact disk player 220 coupled to the I/O bus 210. The VCR 214, DVD 216and compact disk player 220 may be single disk or single cassettedevices, or alternatively may be multiple disk or multiple cassettedevices. They may be independent units or integrated together. Inaddition, the media system 200 includes a microphone/speaker system 222,video camera 224 and a wireless I/O control device 226. According toexemplary embodiments of the present invention, the wireless I/O controldevice 226 is a 3D pointing device according to one of the exemplaryembodiments described below. The wireless I/O control device 226 cancommunicate with the entertainment system 200 using, e.g., an IR or RFtransmitter or transceiver. Alternatively, the I/O control device can beconnected to the entertainment system 200 via a wire.

The entertainment system 200 also includes a system controller 228.According to one exemplary embodiment of the present invention, thesystem controller 228 operates to store and to display entertainmentsystem data available from a plurality of entertainment system datasources and to control a wide variety of features associated with eachof the system components. As shown in FIG. 2, system controller 228 iscoupled, either directly or indirectly, to each of the systemcomponents, as necessary, through I/O bus 210. In one exemplaryembodiment, in addition to or in place of I/O bus 210, system controller228 is configured with a wireless communication transmitter (ortransceiver), which is capable of communicating with the systemcomponents via IR signals or RF signals. Regardless of the controlmedium, the system controller 228 is configured to control the mediacomponents of the media system 200 via a graphical user interfacedescribed below.

As further illustrated in FIG. 2, media system 200 may be configured toreceive media items from various media sources and service providers. Inthis exemplary embodiment, media system 200 receives media input fromand, optionally, sends information to, any or all of the followingsources: cable broadcast 230, satellite broadcast 232 (e.g., via asatellite dish), very high frequency (VHF) or ultra high frequency (UHF)radio frequency communication of the broadcast television networks 234(e.g., via an aerial antenna), telephone network 236 and cable modem 238(or another source of Internet content). Those skilled in the art willappreciate that the media components and media sources illustrated anddescribed with respect to FIG. 2 are purely exemplary and that mediasystem 200 may include more or fewer of both. For example, other typesof inputs to the system include AM/FM radio and satellite radio.

More details regarding this exemplary entertainment system andframeworks associated therewith can be found in the above-incorporatedby reference U.S. patent application “A Control Framework with aZoomable Graphical User Interface for Organizing, Selecting andLaunching Media Items”. Alternatively, remote devices in accordance withthe present invention can be used in conjunction with other systems, forexample computer systems including, e.g., a display, a processor and amemory system or with various other systems and applications.

As mentioned in the Background section, remote devices which operate as3D pointers are of particular interest for the present specification.Such devices enable the translation of movement, e.g., gestures, intocommands to a user interface. An exemplary 3D pointing device 400 isdepicted in FIG. 3. Therein, user movement of the 3D pointing can bedefined, for example, in terms of a combination of x-axis attitude(roll), y-axis elevation (pitch) and/or z-axis heading (yaw) motion ofthe 3D pointing device 400. In addition, some exemplary embodiments ofthe present invention can also measure linear movement of the 3Dpointing device 400 along the x, y, and z axes to generate cursormovement or other user interface commands. In the exemplary embodimentof FIG. 3, the 3D pointing device 400 includes two buttons 402 and 404as well as a scroll wheel 406, although other exemplary embodiments willinclude other physical configurations. According to exemplaryembodiments of the present invention, it is anticipated that 3D pointingdevices 400 will be held by a user in front of a display 408 and thatmotion of the 3D pointing device 400 will be translated by the 3Dpointing device into output which is usable to interact with theinformation displayed on display 408, e.g., to move the cursor 410 onthe display 408. For example, rotation of the 3D pointing device 400about the y-axis can be sensed by the 3D pointing device 400 andtranslated into an output usable by the system to move cursor 410 alongthe y₂ axis of the display 408. Likewise, rotation of the 3D pointingdevice 408 about the z-axis can be sensed by the 3D pointing device 400and translated into an output usable by the system to move cursor 410along the x₂ axis of the display 408. It will be appreciated that theoutput of 3D pointing device 400 can be used to interact with thedisplay 408 in a number of ways other than (or in addition to) cursormovement, for example it can control cursor fading, volume or mediatransport (play, pause, fast-forward and rewind). Input commands mayinclude operations in addition to cursor movement, for example, a zoomin or zoom out on a particular region of a display. A cursor may or maynot be visible. Similarly, rotation of the 3D pointing device 400 sensedabout the x-axis of 3D pointing device 400 can be used in addition to,or as an alternative to, y-axis and/or z-axis rotation to provide inputto a user interface.

According to one purely illustrative exemplary embodiment of the presentinvention, two axes of angular velocity sensing 420 and 422 and threeaxes of linear acceleration sensing 424 can be employed as sensors in 3Dpointing device 400 as shown in FIG. 4. Although this exemplaryembodiment employs inertial sensors it will be appreciated that thepresent invention is not so limited and examples of other types ofsensors which can be used in conjunction with other exemplaryembodiments are provided below. The rotational sensors 420, 422 can, forexample, be implemented using IDG-500 or IXZ-500 sensors made byInvensense. Alternatively, the embodiment could measure all three axesof angular velocity using elements 420 and 422 implemented as thecombination of an IDG-500 and ISZ-500. It will be appreciated by thoseskilled in the art that other types of rotational sensors can beemployed as rotational sensors 420 and 422 and that the Invensensesensors are purely used as an illustrative example. Unlike traditionalgyroscopes, these rotational sensors use MEMS technology to provide aresonating mass which is attached to a frame so that it can resonateonly along one direction. The resonating mass is displaced when the bodyto which the sensor is affixed is rotated around the sensor's sensingaxis. This displacement can be measured using the Coriolis accelerationeffect to determine an angular velocity associated with rotation alongthe sensing axis. Other sensors/sensor packages may also be used, andthe angular velocity sensors 420, 422 can be 1-D, 2-D or 3-D sensors.The accelerometer 424 can, for example, be a 3-axis linear accelerometersuch as the LIS331DLH produced by STMicroelectronics. However, a 2-axislinear accelerometer could be used by assuming that the device ismeasuring gravity and mathematically computing the remaining 3^(rd)value. Additionally, the accelerometer(s) and rotational sensor(s) couldbe packaged together into a single sensor package. Other variations ofsensors and sensor packages may also be used in conjunction with theseexemplary embodiments.

The exemplary embodiments are not limited to the industrial designillustrated in FIGS. 3 and 4, but can instead be deployed in anyindustrial form factor, another example of which is illustrated as FIG.5. In the exemplary embodiment of FIG. 5, the 3D pointing device 500includes a ring-shaped housing 501, two buttons 502 and 504 as well as ascroll wheel 506 and grip 507, although other exemplary embodiments mayinclude other physical configurations. The region 508 which includes thetwo buttons 502 and 504 and scroll wheel 506 is referred to herein asthe “control area” 508, which is disposed on an outer portion of thering-shaped housing 501. More details regarding this exemplaryembodiment can be found in U.S. patent application Ser. No. 11/480,662,entitled “3D Pointing Devices”, filed on Jul. 3, 2006, the disclosure ofwhich is incorporated here by reference.

Such devices have numerous applications including, for example, usage inthe so-called “10 foot” interface between a sofa and a television in thetypical living room as shown in FIG. 6. Therein, as the 3D pointingdevice 500 moves between different positions, that movement is detectedby one or more sensors within 3D pointing device 200 and transmitted tothe television 620 (or associated system component, e.g., a set-top box(not shown)). Movement of the 3D pointing device 500 can, for example,be translated or mapped into movement of a cursor 640 displayed on thetelevision 620 (examples of such mappings being provided below) andwhich is used to interact with a user interface. Details of an exemplaryuser interface with which the user can interact via 3D pointing device500 can be found, for example, in the above-incorporated U.S. patentapplication Ser. No. 10/768,432 as well as U.S. patent application Ser.No. 11/437,215, entitled “Global Navigation Objects in User Interfaces”,filed on May 19, 2006, the disclosure of which is incorporated here byreference. Another exemplary embodiment may contain other sensorsincluding the sensing system which can be found in U.S. patentapplication Ser. No. 12/424,090 entitled “Tracking Determination BasedOn Intensity Angular Gradient Of A Wave”, filed on Apr. 15, 2009, thedisclosure of which is incorporated here by reference.

One challenge faced in implementing exemplary 3D pointing devices 400 inaccordance with these exemplary embodiments is to employ components,e.g., rotational sensors 502 and 504, which are not too costly, while atthe same time providing a high degree of correlation between movement ofthe 3D pointing device 400, a user's expectation regarding how the userinterface will react to that particular movement of the 3D pointingdevice and actual user interface performance in response to thatmovement. For example, if the 3D pointing device 400 is not moving, theuser will likely expect that the cursor ought not to be drifting acrossthe display. Likewise, if the user rotates the 3D pointing device 400purely around the y-axis, she or he would likely not expect to see theresulting cursor movement on display 408 contain any significant x-axiscomponent. To achieve these, and other, aspects of exemplary embodimentsof the present invention, various measurements and calculations areperformed, e.g., by the handheld device 400, which are used to adjustthe outputs of one or more of the sensors 420, 422 and 424 and/or aspart of the input used by a processor to determine an appropriate outputfor the user interface based on the outputs of the sensors 420, 422 and424. These measurements and calculations are used to compensate forfactors which fall broadly into two categories: (1) factors which areintrinsic to the 3D pointing device 400, e.g., errors associated withthe particular sensors 420, 422 and 424 used in the device 400 or theway in which the sensors are mounted in the device 400 and (2) factorswhich are not intrinsic to the 3D pointing device 400, but are insteadassociated with the manner in which a user is using the 3D pointingdevice 400, e.g., linear acceleration, tilt and tremor. Some exemplarytechniques for handling these effects are described in theabove-incorporated by reference '518 patent. However, additionaltechniques for handling the bias or offset error contributions to sensedmotion are described in U.S. patent application Ser. No. 12/163,229filed on Jun. 27, 2008, entitled “Real-Time Dynamic Tracking Of Bias”,the disclosure of which is incorporated here by reference.

Mapping

As mentioned above, a 3D pointing device that a user moves in 6 degreesof freedom can be used to convert that motion into cursor motion.Different applications have different demands and requirements on howthe device motion should be mapped into cursor motion. These exemplaryembodiments describe novel methods for mapping the device motion intocursor motion which provide improved 3D pointer performance and areconfigurable to deliver optimal response for each application. Amongother things, exemplary embodiments of the present invention describenew 3D pointer mapping methods and a method for combining alternatemapping methods to provide optimal response for a given application.These can, for example, reduce cursor resolution variation as a functionof the user's position relative to the display, which is a primaryproblem with absolute pointing. At the same time, the exemplaryembodiments can provide a constant mapping between the device positionand cursor position, which can be a problem for some relative pointingapplications.

One exemplary embodiment includes:

1. A device that measures motion in 6 degrees of freedom

2. One or more algorithms that convert the device motion into cursormotion

3. A method that combines the cursor motion outputs from the algorithms

4. A visual display that displays the cursor

In the exemplary embodiment shown in FIG. 7, a user holds the device 701and makes motions to control the cursor 703 which appears on display702. The device 701 typically contains a collection of sensors, examplesof which were described above. The sensing system may consist of one ormore sensors including linear accelerometers, angular position sensors(traditional gyroscopes), angular velocity sensors (MEMS gyroscopes),magnetometers, cameras, optical, ultrasonic and RF. The sensing systemprocesses the sensor data to provide an estimate of the device's linearposition and angular position. The device's position is then processedby one or more mapping algorithms to yield cursor locations. The cursorlocations are then combined to produce a final cursor output. The cursormotion information then drives the cursor on the display 702. The device701 may be connected via either wires or wirelessly to the display 702.The algorithm may run on the device 701, the display 702 or anintermediate processing unit (not shown in FIG. 7, e.g., a systemconsole connected to both the device 701 and the display 702).

According to one exemplary embodiment, the device 701 is abattery-powered, handheld device which contains a 3-axis accelerometerand a 3-axis gyroscope, however it will be appreciated that fewer orother sensors may be included. According to this exemplary embodiment,the device 701 processes the sensor data to estimate its linear positionand angular position and further processes the linear position andangular position to produce the cursor motion data. The cursor motiondata is communicated to a set-top box, e.g., represented by systemcontroller 228 in FIG. 2, over a proprietary 2.4 GHz RF link. The datais received by the RF hardware in the set-top box and communicated overa USB bus to the main set-top box processor. The main set-top boxprocessor moves the cursor as specified, in this example, by the device701. At 30 Hz or 60 Hz for NTSC (25 Hz or 50 Hz for PAL), the set-topbox renders the image and sends the image to the display over, e.g.,HDMI, component, S-Video, and/or composite outputs. The display receivesthe image and displays the image to the user. As mentioned earlier,although the various processing stages are performed in the handhelddevice 701 according to this exemplary embodiment, the set-top box orother controller which communicates with the handheld device 701 couldperform some or all of the processing, e.g., the estimation of linearposition and angular position and/or the mapping of the estimated linearposition and angular position into one or more cursor locations.

Prior to discussing the mapping techniques which can be employedaccording to exemplary embodiments, some mathematical notation isintroduced below to guide the discussion:

Lower case letters represent scalar variables: x, y, z;

Lower case bold letters represent vectors: x, y, z;

Upper case bold letters represent matrices: X, Y, Z;

Vectors are assumed to be column vectors (N×1 matrix);

|v| is the magnitude of vector v;

x y=x·y is the dot product of vectors x and y;

x×y is the cross product of vectors x and y;

X y is the matrix multiplication of matrix X and vector y;

X^(T) is the matrix transpose;

ŷ is the unit vector in the direction of

${y = \frac{y}{\sqrt{y \cdot y}}};$

<q₀, v> is the quaternion q with scalar component q₀ and the length 3vector v;

Vector(q)=v where q is the quaternion <q₀, v>;

q

p is the quaternion multiplication of q and p;

q* is the quaternion conjugate of q: <q₀, v>*=<q₀, −v>;

^(b)x is the vector x defined in body-frame coordinates;

^(u)x is the vector x defined in user-frame coordinates;

Length 2 vector v is assumed to have subcomponents named (v_(x), v_(y));and

Length 3 vector v is assumed to have subcomponents named (v_(x), v_(y),v_(z)).

Using this notation, exemplary techniques for processing device motioninto cursor movement as shown in FIG. 7 will now be considered. Thefigure shows two coordinate systems. A first coordinate system, i.e.,the user-frame coordinate system, is arbitrarily chosen in this exampleto be the center of the display and consists of (x, y, z). Theuser-frame coordinate system is stationary with respect to the display.The coordinate system has x into the display, y to the right of thedisplay and z down, which corresponds to a typical aerospace coordinatesystem convention. Those skilled in the art will recognize thatalternate conventions could be used instead and include, for example, PCdisplay coordinates (x right, y down, z into display) and HID (x out ofdisplay, y right, z down). For this discussion, the user-frame axes willbe arbitrarily defined as:

^(u)x=[1, 0, 0]

^(u)y=[0, 1, 0]

^(u)z=[0, 0, 1]

One skilled in the art will recognize that the user-frame coordinatesystem axes and origin can be chosen without materially altering thepresent invention. The discussion above assumes a Cartesian coordinatesystem, but other coordinate systems, such as spherical coordinates,could also be used without affecting the invention.

The second coordinate system in this exemplary embodiment is thedevice's body-frame coordinate system. The body-frame coordinate systemis stationary with respect to the device. The body-frame origin istypically at the center of the device, although that is not required.The body-frame axes are shown as (x°, y°, z°) with x° out the front ofthe device, y° to the right, and z° down. For this discussion, thebody-frame axes are arbitrarily defined as:

^(b)x°=[1, 0, 0]

^(b)y°=[0, 1, 0]

^(b)z°=[0, 0, 1]

One skilled in the art will recognize that the body-frame coordinatesystem axes and origin can be chosen without materially altering thepresent invention. The discussion above assumes a Cartesian coordinatesystem, but other coordinate systems, such as spherical coordinates,could also be used without affecting the invention.

The length 3 vector ou is the origin of the user-frame coordinate systemand is defined to be fixed to the display. ^(u)ou is defined as (0, 0,0) for the present embodiment. The length 3 vector ob is the origin ofthe body-frame coordinate system and is defined to be fixed to thedevice. ^(b)ob is defined as (0, 0, 0) for the present embodiment. Thelength 3 vector od is the origin of the display coordinate system and isdefined to be fixed to the display. ^(u)od is defined as (0, c_(x),c_(y)) for the present embodiment. The length 3 vector u is defined asthe device's linear position in 3-D space. u=ob. The length 3 vector wis the vector from the device's body-frame coordinate system origin, ob,to the user-frame coordinate system origin, ou. Mathematically,w=ou−ob=−u.

The quaternion q is defined as the device's angular position withrespect to user-frame coordinates. For the present discussion, q isdefined as a unit quaternion (q·q=1). Instead of quaternions, anydesired alternate angular position representation could also be usedincluding, for example, Euler angles, direction cosine matrix (DCM), andvector/angle. The length 2 vector p is the pointing device's cursorlocation on the display given as 2-D display coordinates. The length 2vector p₀ is the cursor location of the display coordinate system. Thelength 3 vector ω is the angular velocity. Following the conventionabove, ^(u)ω is the angular velocity in user-frame coordinates and ^(b)ωis the angular velocity in the device's body-frame coordinates. Given avector in body-frame coordinates, the vector in user-frame coordinatescan be found using the following equation:

^(u) v=Vector(q

0,^(b) v

q*)

Given a vector in user-frame coordinates, the vector in body-framecoordinates with the following equation:

^(b) v=Vector(q*

0,^(u) v

q)

Given this exemplary, mathematical context, mapping device motion intocursor location using absolute pointing will first be considered. Thecursor in an absolute pointing system should be located at theintersection of the display and the line containing the device's x°axis. For example, if a laser pointer were appropriately mounted to thedevice 701, the cursor location and the laser pointer's dot on thedisplay 702 should at the same position. The full mathematical equationfor absolute pointing given user-frame coordinates is:

$p = {{- {\frac{{}_{}^{}{}_{}^{}}{{\,^{u}x}\square_{x}}\begin{bmatrix}{{\,^{u}x}\square_{y}} \\{{\,^{u}x}\square_{z}}\end{bmatrix}}} + \begin{bmatrix}{{}_{}^{}{}_{}^{}} \\{{}_{}^{}{}_{}^{}}\end{bmatrix} + p_{0}}$

Absolute pointing is especially suitable for shooting games where theuser aims directly at targets on the display. Absolute pointing isanalogous to how humans point our arms, hands and fingers to physicalobjects in our environment. With absolute pointing, the ratio of cursormotion to angular device motion changes as a function of distance. Thefurther the device is from the display, the smaller the angular motionsrequired to move the cursor a given distance on the display. Likewise,the further the device is from the x axis, the smaller the angularmotions required to move the cursor. Thus, for example, an absolutepointing algorithm maps device motion to cursor location based upon anintersection of a forward pointing direction (body-frame x-axis) of thedevice and a surface of a display on which the cursor is to bedisplayed.

The value of ^(u)u_(x) corresponds to a distance correction factor. Thisdistance correction factor is described by U.S. Pat. No. 5,627,565entitled “Space coordinates detecting device and input apparatus usingsame”, the disclosure of which is incorporated here by reference.

Next, relative pointing or “body-frame” relative pointing will beconsidered. Some applications do not required direct pointing at adisplay or may not have any meaningful frame of reference. For suchcases, the application may choose to use the device itself as theprimary reference. As long as the ^(u)x°_(x) is positive, regardless ofthe position of the display or the position of the device, if the usermoves the device linearly along the y° axis or rotates around the z°axis, the cursor will always move to the right.

Assuming discrete time with Euler integration, body-frame relativepointing is mathematically defined by:

${\Delta \; p} = {{\begin{bmatrix}{{}_{}^{}{}_{}^{}} \\{{}_{}^{- b}{}_{}^{}}\end{bmatrix}\Delta \; t} - \begin{bmatrix}{\Delta {{}_{}^{}{}_{}^{}}} \\{\Delta {{}_{}^{}{}_{}^{}}}\end{bmatrix}}$

Note that devices need not implement both terms in this equation.Relative pointing is illustrated functionally in FIG. 8. With relativepointing, unlike absolute pointing, the amount of cursor motion is notaffected by the position of the device 800. However, the forwarddirection of the device and the cursor location are not deterministicand may vary over time. For some applications, this decoupling isbeneficial. The actual cursor location may be computed using anon-linear function of Δp, referred to as vPointer in FIG. 8, which is anon-linear function often called pointer ballistics 802. Traditional 2-Dcomputer mice use pointer ballistics 802 to improve their apparentusability and performance, and this function can also be used in 3Dpointer mapping. The output of the pointer ballistics 802, vSys, is thenused to move the cursor on screen subject to boundary conditions, etc.,in block 804.

Although body-frame relative pointing offers some advantages, usersoften do not care about the position of the device, only the relativemovements that they make. Thus, a third type of mapping which isconsidered here is referred to as “user-frame relative pointing”. Inuser-frame relative pointing, if the user rotates the device around theuser-frame z-axis then the cursor should move to the right. If the userlinearly moves the device along the user-frame z-axis, then the cursorshould move down.

Assuming discrete time with Euler integration, user-frame relativepointing is mathematically defined by:

 ^(b)z = Vector(q^(*) ⊗ ⟨0,  ^(u)z⟩ ⊗ q)$\theta = {\tan^{- 1}\left( \frac{{}_{}^{}{}_{}^{}}{{}_{}^{}{}_{}^{}} \right)}$${\Delta \; p} = {{{\begin{bmatrix}{\cos \; \theta} & {\sin \; \theta} \\{{- \sin}\; \theta} & {\cos \; \theta}\end{bmatrix}\begin{bmatrix}{{}_{}^{}{}_{}^{}} \\{{}_{}^{- b}{}_{}^{}}\end{bmatrix}}\Delta \; t} - \begin{bmatrix}{\Delta {{}_{}^{}{}_{}^{}}} \\{\Delta {{}_{}^{}{}_{}^{}}}\end{bmatrix}}$

Note that the device need not implement both terms. U.S. Pat. No.7,158,118, the disclosure of which is incorporated by reference above,contains details and disclosures regarding user-frame relative pointing.In one exemplary embodiment described therein, −^(b)z is measured by anaccelerometer. As with body frame relative pointing, with user framerelative pointing the amount of cursor motion is not affected by theposition of the device. However, the forward direction of the device andthe cursor location are not deterministic and may vary over time. Forsome applications, this decoupling is beneficial. The actual cursorlocation may be computed using a non-linear function of Δp, often calledpointer ballistics. Traditional 2-D computer mice use pointer ballisticsto improve their apparent usability and performance.

For many applications, the device responsiveness should be constantregardless of its position within the room but should still be pointingtowards the display. According to exemplary embodiments, a fourthmapping technique, referred to herein as “absolute invariant pointing”blends many of the relative pointing benefits with the absolute pointingbenefits while minimizing the negative factors of each method. Absoluteinvariant pointing can be mathematically defined by:

 ^(u)x▫ = Vector(q ⊗ ⟨0,  ^(u)x▫⟩ ⊗ q^(*)) $p = {{{c\begin{bmatrix}{\tan^{- 1}\frac{{\,^{u}x}\square_{y}}{{\,^{u}x}\square_{x}}} \\{\tan^{- 1}\frac{{\,^{u}x}\square_{z}}{{\,^{u}x}\square_{x}}}\end{bmatrix}} + \begin{bmatrix}{{}_{}^{}{}_{}^{}} \\{{}_{}^{}{}_{}^{}}\end{bmatrix} + p_{0}} \approx {{c\begin{bmatrix}{\sin^{- 1}{\,^{u}x}\square_{y}} \\{\sin^{- 1}{\,^{u}x}\square_{z}}\end{bmatrix}} + \begin{bmatrix}{{}_{}^{}{}_{}^{}} \\{{}_{}^{}{}_{}^{}}\end{bmatrix} + p_{0}}}$

where c is a constant.

Absolute invariant pointing, like absolute pointing per se describedabove, contains a direct, repeatable mapping from device linear positionand angular position into cursor location. At the same time, the cursorresponsiveness to linear motion and angular motion is consistent overthe motion range. If the device is located to the left or right ofdisplay center, then users typically attempt to move the device left orright relative to the vector from the device to the display, not thedisplay normal vector, x. This results in an apparent loss of linearmotion resolution as the device moves away from the x-axis. Thus, anabsolute invariant mapping algorithm according to this exemplaryembodiment generates cursor position as a sum of a term of linearposition values and a term computed from angular position that isindependent of linear position.

A fifth mapping technique called “virtual display” reduces the loss ofapparent linear resolution found with “absolute invariant pointing” asthe device moves off-axis. The “virtual display” technique may create anintermediate, virtual display that moves to face the device. By movingto face the device, the virtual display maintains the same resolution asif the device was directly in front of the display. A full virtualdisplay moves to directly face the device. A new virtual displaycoordinate system is created by construction with axes x^(Δ), y^(Δ), andz^(Δ). In mathematical terms,

^(u) x ^(Δ)=−^(u) û

Typically, the new coordinate system will preferably, but not as arequirement, maintain the y-axis as “horizontal” and the z-axis as“vertical”. By construction, the remaining axes will then be:

b= ^(u) z× ^(u) x ^(Δ)

^(u)y^(Δ)={circumflex over (b)}

^(u) z ^(Δ)=^(u) x ^(Δ)×^(u) y ^(Δ)

By fully rotating the virtual display to face the device, the cursorresponse becomes highly non-linear as the device points away from theorigin. To help minimize these effects, the virtual display coordinatesystem need not fully rotate towards the device. A scaling factor can beapplied to construct ^(u)x^(Δ) between ^(u)x and ^(u)û. For example,

^(u) x ^(Δ) =c ^(u) x−(1−c)^(u) û

where c is a value between 0 and 1, inclusive.

The virtual display can be applied to the absolute pointing algorithm tocreate a planar virtual display:

$Q = \begin{bmatrix}| & | & | \\{{}_{}^{}{}_{}^{}} & {{}_{}^{}{}_{}^{}} & {{}_{}^{}{}_{}^{}} \\| & | & |\end{bmatrix}$ s = Q ^(u)x▫w = Q ^(u)u$p = {{\frac{w_{x}}{s_{x}}\begin{bmatrix}s_{y} \\s_{z}\end{bmatrix}} + \begin{bmatrix}w_{y} \\w_{z}\end{bmatrix} + p_{0}}$

The virtual display method can also be applied to any absolute orpseudo-absolute mapping method including the absolute pointing andabsolute invariant pointing described above. If the cursor resides on anon-planar display, then this method could easily be adapted to create avirtual display of the non-planar display. Thus, according to exemplaryembodiments, the intermediate, planar virtual display maps device motionto cursor location based upon an intersection of a forward pointingdirection (body-frame x-axis) of the device and a surface of a displayon which the cursor is to be displayed rotated to at least partiallyface the device.

For many applications, angular response is more important than linearresponse. However, the virtual display method does not have consistentangular response. The angular responsiveness is similar to absolutepointing, not relative pointing.

A sixth mapping technique called “virtual spherical display” maintainsconstant angular response, unlike the “virtual display”. For example, ifa conversion to polar coordinates is first performed, i.e.:

$b = {{Q{\,^{u}x}\square} = \begin{bmatrix}{\cos \; \varphi \; \cos \; \theta} \\{\cos \; \varphi \; \sin \; \theta} \\{{- \sin}\; \varphi}\end{bmatrix}}$

Then, the algorithm can solve for θ and φ and finally p using homogenouscoordinates as follows:

φ = sin⁻¹(−b_(z))$\theta = {{{\tan^{- 1}\left( \frac{b_{y}}{b_{x}} \right)}\begin{bmatrix}p_{x} \\p_{y} \\1\end{bmatrix}} = {T\begin{bmatrix}\theta \\{- \varphi} \\1\end{bmatrix}}}$

where T is a 3×3 general transformation matrix of arbitrary constants.The matrix T may apply any combination of a scale, rotation,translation, shearing, reflection, orthogonal projection, affinetransformation or perspective projection. Thus, the spherical virtualdisplay according to this exemplary embodiment maps device motion tocursor location based upon spherical coordinates of the estimatedangular position being transformed into the virtual display coordinatesby a transformation matrix and converted into cursor location.

The spherical virtual display method can be applied to relative pointingto create a seventh mapping technique called “relative spherical virtualdisplay”. Instead of controlling the cursor using angles, this methoduses the change in the angle to drive a change in the cursor.

$\begin{bmatrix}{\Delta \; p_{x}} \\{\Delta \; p_{y}} \\1\end{bmatrix} = {T\begin{bmatrix}{{{}_{}^{}{}_{}^{}}\Delta \; t} \\{{- {{}_{}^{}{}_{}^{}}}\Delta \; t} \\1\end{bmatrix}}$

where T is a 3×3 general transformation matrix of arbitrary constants.The matrix T may apply any combination of a scale, rotation,translation, shearing, reflection, orthogonal projection, affinetransformation or perspective projection. Thus, the intermediate,relative, spherical virtual display according to this exemplaryembodiment play maps device motion to cursor location based uponbody-frame angular velocity modified by a transformation matrix andconverted into cursor change in location.

The angular position can also be mapped into cursor coordinates using aspherical projection. This eighth mapping method is known as “angularposition spherical projection”.

$b = {{{q_{I}^{*} \otimes q^{*} \otimes {\,^{u}x}}{\square{\otimes {q \otimes q_{I}}}}} = \begin{bmatrix}{\cos \; \varphi \; \cos \; \theta} \\{\cos \; \varphi \; \sin \; \theta} \\{{- \sin}\; \varphi}\end{bmatrix}}$

where q_(I) is an arbitrary initial angular position rotation value.Then, the algorithm can solve for θ and φ and finally p using homogenouscoordinates as follows:

φ = sin⁻¹(−b_(z))$\theta = {{{\tan^{- 1}\left( \frac{b_{y}}{b_{x}} \right)}\begin{bmatrix}p_{x} \\p_{y} \\1\end{bmatrix}} = {T\begin{bmatrix}\theta \\{- \varphi} \\1\end{bmatrix}}}$

where T is a 3×3 general transformation matrix of arbitrary constants.The matrix T may apply any combination of a scale, rotation,translation, shearing, reflection, orthogonal projection, affinetransformation or perspective projection. One skilled in the art willnote that if the third row of p in homogenous coordinates is not equalto one, then p can be scaled so that it becomes one.

The outputs of all or some of the absolute methods described above canbe joined by simple linear combination or a more complicated non-linearcombination. For example, suppose that an application wanted to bemostly absolute but with improved angular resolution pointingconsistency. In this case, a combination of the above described“absolute pointing mapping” with the “invariant absolute pointingmapping” can be performed. Exemplary embodiments can, for example, use asimple linear combination (or a more complicated non-linear process) tocombine any two more of the described methods. For each method, p_(i),an application assigns a weighting factor c_(i). The final resulting pis then:

$p = {\sum\limits_{i}{c_{i}p_{i}}}$

Ideally,

${\sum\limits_{i}c_{i}} = 1$

to maintain a consistent cursor response, but this constraint is notrequired.

Similarly, the outputs of all or some of the relative methods describedabove can be joined by simple linear combination or a more complicatednon-linear combination. For example, suppose that both the angularposition and angular velocity of a device had unique flaws. Anembodiment could use an equal linear combination of “user-frame relativepointing” with “relative spherical virtual display” which would reducethe flaws of each method by half The exemplary embodiment uses a simplelinear combination to combine the methods. For each method, p_(i), anapplication assigns a weighting factor c_(i). The final resulting Δ p isthen:

${\Delta \; p} = {\sum\limits_{i}{c_{i}\Delta \; p_{i}}}$

Ideally,

${\sum\limits_{i}c_{i}} = 1$

to maintain a consistent cursor response, but this constraint is notrequired.

Combining absolute pointing methods and relative pointing methods isalso considered according to these exemplary embodiments. One method inwhich combined absolute and relative pointing mappings can be employedis to have the cursor primarily controlled through relative pointing,but use absolute pointing to adjust the cursor movement so as to avoidlong-term drift from the reference. Avoiding long-term drift wouldeliminate the periodical re-centering the cursor typical with relativepointing solutions. When using relative pointing and when usingnon-linear pointer ballistics according to an exemplary embodiment, themapping between 3D pointer position and cursor position is time-varyingand dependent upon the position and range of motion, and also the speedof movement. By retaining history of 3D pointer positions and theirassociated cursor locations, it is possible to define an adaptive mapthat defines a cursor position based upon a 3D pointer position. Thismap can be constrained to minimize the difference between the history ofcursor positions and the history of 3D pointer positions mapped tocursor positions with the map.

An example of such an embodiment is illustrated in FIG. 9. In thisparticular exemplary embodiment, the “user-frame relative pointing”mapping is combined with the “angular position spherical projection”,both of which mappings are individually described above. The pointingdevice 900 outputs, on the lower branch 902, vPointer data which is theuser-frame relative pointing data described above with respect to FIG.8. On the upper branch 904, the pointing device 900 outputs angularposition which is used by the absolute pointing mapping algorithm as aninput. A map is created in block 906 which, in this exemplaryembodiment, includes a 3×3 general transformation matrix T which canperform a variety of transformations on the output angular position,e.g., scale (stretch in any of the axes), rotation (preserveorthogonality), shearing (essentially make axes non-orthogonal),translation (offset to account for 2D application of data), reflection,and any other affine transformation or perspective projection. The mapalso defines an origin value (quaternion). A detailed example of how themap can be created is shown in FIG. 10.

Therein, in block 1002, the angular position output from the device isfirst rotated to map the measured angular position of the device toaccount for deviations from the nominal origin. After applying theinitial rotation, the angular position is converted to sphericalcoordinates at block 1004. The current sample is evaluated to determinea weight for the sample at block 1006. The weights capture how importanteach previous point is to defining the current map between where thecursor is currently located on the screen and the angular position ofthe pointing device 900. The weights assist with determining which datapoints are worth saving and can be used as part of the least squaressolution to find the map. In the exemplary implementation, the primaryweight is applied to each sample based upon the absolute angularposition of the pointing device. The full range of motion of the deviceis divided into a fixed set of small regions. The first data point ineach region gets the largest weight, and every future point within thatregion gets a smaller weight. In addition a secondary weight based onthe current angular velocity is applied so that points where the deviceis at rest are more important than for points where the device is inmotion. Based on this weight, the best N samples of cursor position,angular position, and weighted samples are saved and used for mapcalculation at blocks 1008, 1010 and 1012, respectively. The map iscreated at block 1014 by first calculating the rotation origin to ensurethat the input data remains within a reasonable range for conversion tospherical coordinates. If the cursor has moved beyond the displaybounds, the origin and the saved state can be adjusted to define a newalignment. Next a 3×3 general transformation matrix T is created thatwill transform the set of azimuth/elevation pairs onto the set of cursorpositions. One method of finding T is to define the error vector v_(n)for each of the N saved samples as:

$v_{n} = {{T\begin{bmatrix}\theta_{n} \\{- \varphi_{n}} \\1\end{bmatrix}} - \begin{bmatrix}x_{n} \\y_{n} \\1\end{bmatrix}}$

and then minimize the least squares error

$\underset{T}{\arg \; \min}\left\lbrack {\sum\limits_{n = 1}^{N}{w_{n}\left( {v_{n} \cdot v_{n}} \right)}} \right\rbrack$

where w_(n) is the weight for each sample. One skilled in the art willrecognize that many methods exist for finding a solution to this linearleast squares problem including inverting the normal equations using theMoore-Penrose pseudoinverse and orthogonal decomposition methods such asQR decomposition or singular value decomposition. QR decomposition isused in the exemplary implementation.

Returning to FIG. 9, once a map is defined by block 906 it is used tomap the current 3D pointer position to display coordinates to generate areference cursor location pRef in block 908. More details regarding thisexemplary process are illustrated in FIG. 11. Therein, at block 1102,the angular position of the device 900 is rotated in the same way asdescribed above with respect to block 1002, and the output is translatedinto spherical coordinates by block 1104 to create the 2×1 vector calledAz/El. This is then converted to homogenous coordinates and thenmultiplied with the map matrix T at block 1106 yielding pRef. The pRefvalue thus represents the desired location for the cursor based upon theabsolute pointing system component of this exemplary, combined mapping.

Returning again to FIG. 9, the relative pointing value vSys, theabsolute pointing value pRef and the current cursor position p0, areinput to a dynamic ballistics function 910. The dynamic ballisticsfunction 910 takes these inputs and, in this exemplary embodiment, usesthe absolute pointing value and the current cursor position to adjustthe relative pointing value. More specifically, as shown in FIG. 12, thecurrent and reference cursor positions are used to adjust the cursormovement before it is applied to the cursor. One method of adjusting thecursor is to perform small adjustments to the scale and angle of thecurrent velocity vector so that the new cursor position will be closerto the reference point. First, the current point is subtracted from thereference point to get the reference velocity vector vRef at block 1202.This reference vector is compared to the original velocity v0 to findthe angle between the vectors at block 1204. This angle is limited to afixed maximum, at block 1206, and then used to rotate the vector v0 tocreate vRot as shown in block 1208 of FIG. 12. Next, the referencevector is projected onto vRot to find the scaling of vRot that would getthe next point closest to the reference point as shown in block 1210.This scaling is limited between a maximum and minimum value (block 1212)and then applied to vRot at block 1214. The limits on maximum angle andmaximum scale can be tuned to control how much correction will beapplied.

By combining absolute pointing methods and relative pointing methods itis possible to maintain the benefits of relative pointing but stillmaintain a fixed region of 3D device positions that map to the display.This allows for drift-less relative pointing, which has the invariantthat given a 3D pointer position q and cursor location p1 at time t₀,returning the device to position q at time t₁>t₀ will return the cursorlocation to p2 such that p1≈p2.

Numerous variants and alternatives of the foregoing mapping techniquesare contemplated. For example, the combination need not be limited tofull mapping methods. Each term of each mapping method could be assignedits own weighting factor. Moreover, the virtual display need not fullytrack the device. Compromises include:

-   -   1. Adding (or subtracting) an arbitrary x-axis value to the        actual device position to appropriately scale linear motion.    -   2. Algorithmically increasing or decreasing the distance to the        virtual display from the device.    -   3. Multiplying the device's actual position by a scale factor to        reduce or increase the response to linear motion.    -   4. A combination of any of the above factors.        Alternative virtual display surfaces could be used. The above        text describes planes and spheres, but the virtual display could        be extended to cylinders, ellipses and higher order surfaces.

Thus according to one exemplary embodiment, a method for mapping adevice's movement into cursor position can include the steps illustratedin the flowchart of FIG. 13. Therein, at step 1300, at least one of a 3Dpointing device's linear position and angular position can be estimated(or sensed, measured, detected, etc.). In some cases, exemplary mappingalgorithms described above may use only a device's linear position, onlya device's angular position or both its linear position and angularposition as inputs to the mapping algorithms. Then, at step 1302, atleast one of the estimated linear position and estimated angularposition are processed using both a first mapping algorithm, to generatea first cursor location, and a second mapping algorithm, to generate asecond cursor location. The results are combined at step 1304 togenerate a final cursor output.

The mapping algorithms may operate on partial or incomplete motion data.For example, user-frame relative pointing is useful with only one of thetwo terms. Some applications can utilize user-frame relative pointingwhile collecting either angular motion or linear motion, but not both.Sensor(s) may collect and estimate motion in either user-framecoordinates, body-frame coordinates or a combination of user-frame andbody-frame coordinates. The mapping algorithm may operate in either thebody-frame coordinate system, user-frame coordinate system, or any othercoordinate system. Motion may be measured in any coordinate systemincluding Cartesian and spherical. The mapping algorithm may usederivatives of linear position including velocity and acceleration. Themapping algorithm may use derivatives of angular position includingvelocity and acceleration. The mapping combination method may be trivialand only use data from one mapping method. The factors for otheralgorithms may be 0. Mathematical terms with 0 valued coefficients neednot be computed or appear in the final implementation.

Systems and methods for processing data according to exemplaryembodiments of the present invention can be performed by one or moreprocessors executing sequences of instructions contained in a memorydevice. Such instructions may be read into the memory device from othercomputer-readable mediums such as secondary data storage device(s).Execution of the sequences of instructions contained in the memorydevice causes the processor to operate, for example, as described above.In alternative embodiments, hard-wire circuitry may be used in place ofor in combination with software instructions to implement the presentinvention. Such software may run on a processor which is housed withinthe device, e.g., a 3D pointing device or other device, which containsthe sensors or the software may run on a processor or computer housedwithin another device, e.g., a system controller, a game console, apersonal computer, etc., which is in communication with the devicecontaining the sensors. In such a case, data may be transferred viawireline or wirelessly between the device containing the sensors and thedevice containing the processor which runs the software which performsthe pointer mapping as described above. According to other exemplaryembodiments, some of the processing described above with respect topointer mapping may be performed in the device containing the sensors,while the remainder of the processing is performed in a second deviceafter receipt of the partially processed data from the device containingthe sensors.

Although the foregoing exemplary embodiments relate to sensing packagesincluding one or more rotational sensors and an accelerometer, pointermapping techniques according to these exemplary embodiments are notlimited to only these types of sensors. Instead pointer mappingtechniques as described herein can be applied to devices which include,for example, only accelerometer(s), optical and inertial sensors (e.g.,a rotational sensor, a gyroscope, an angular velocity sensor or a linearaccelerometer), a magnetometer and an inertial sensor (e.g., arotational sensor, a gyroscope or a linear accelerometer), amagnetometer and an optical sensor, or other sensor combinations.Additionally, although exemplary embodiments described herein relate tocursor mapping in the context of 3D pointing devices and applications,such techniques are not so limited and may be employed in methods anddevices associated with other applications, e.g., medical applications,gaming, cameras, military applications, etc.

The above-described exemplary embodiments are intended to beillustrative in all respects, rather than restrictive, of the presentinvention. Thus, the present invention is capable of many variations indetailed implementation that can be derived from the descriptioncontained herein by a person skilled in the art. For example, althoughthe foregoing exemplary embodiments describe, among other things, theuse of inertial sensors to detect movement of a device, other types ofsensors (e.g., ultrasound, magnetic or optical) can be used instead of,or in addition to, inertial sensors in conjunction with theafore-described signal processing. All such variations and modificationsare considered to be within the scope and spirit of the presentinvention as defined by the following claims No element, act, orinstruction used in the description of the present application should beconstrued as critical or essential to the invention unless explicitlydescribed as such. Also, as used herein, the article “a” is intended toinclude one or more items.

1. A method for mapping a device's movement into cursor position,comprising: estimating at least one of said device's linear position andangular position; processing at least one of said estimated linearposition and estimated angular position using both a first mappingalgorithm to generate a first cursor location and a second mappingalgorithm to generate a second cursor location; and combining said firstcursor location and said second cursor location to generate a finalcursor output.
 2. The method of claim 1, wherein said first mappingalgorithm is an absolute pointing algorithm and wherein said secondmapping algorithm is a relative pointing algorithm.
 3. The method ofclaim 1, wherein at least one of the mapping algorithms is an absolutepointing algorithm which has a characteristic that a ratio of cursormotion to angular device motion changes as a function of distancebetween said device and a display on which a cursor is displayed.
 4. Themethod of claim 3, wherein said absolute pointing algorithm maps devicemotion to cursor location based upon an intersection of a forwardpointing direction (body-frame x-axis) of the device and a surface of adisplay on which said cursor is to be displayed.
 5. The method of claim4, wherein said absolute pointing algorithm is defined as:${p = {{- {\frac{{}_{}^{}{}_{}^{}}{{\,^{u}x}\square_{x}}\begin{bmatrix}{{\,^{u}x}\square_{y}} \\{{\,^{u}x}\square_{z}}\end{bmatrix}}} + \begin{bmatrix}{{}_{}^{}{}_{}^{}} \\{{}_{}^{}{}_{}^{}}\end{bmatrix} + p_{0}}},$ where ^(u)u_(x) is a detected value of thelinear position x-axis in a user's frame of reference; ^(u)u_(y) is adetected value of the linear position y-axis in a user's frame ofreference; ^(u)u_(z) is a detected value of the linear position z-axisin a user's frame of reference; ^(u)x°_(x) is a detected value of thebody-frame x-axis in a user's frame of reference; ^(u)x°_(y) is adetected value of the body-frame y-axis along an y-axis in a user'sframe of reference; ^(u)x°_(z) is a detected value of the body-framez-axis along a z-axis in a user's frame of reference; and p₀ is the 2Dvector value of the display coordinate system origin in cursorcoordinates.
 6. The method of claim 1, wherein at least one of themapping algorithms is an absolute invariant algorithm which has acharacteristic of providing a direct, repeatable mapping from devicelinear position and angular position into cursor location and a secondcharacteristic that cursor responsiveness to linear motion and angularmotion is consistent over a motion range.
 7. The method of claim 6,wherein said absolute invariant algorithm generates cursor position as asum of a term of linear position values and a term computed from angularposition that is independent of linear position.
 8. The method of claim7, wherein said absolute invariant algorithm is defined as: ^(u)x▫ = Vector(q ⊗ ⟨0,  ^(b)x▫⟩ ⊗ q^(*)) $p = {{c\begin{bmatrix}{\tan^{- 1}\frac{{\,^{u}x}\square_{y}}{{\,^{u}x}\square_{x}}} \\{\tan^{- 1}\frac{{\,^{u}x}\square_{z}}{{\,^{u}x}\square_{x}}}\end{bmatrix}} + \begin{bmatrix}{{}_{}^{}{}_{}^{}} \\{{}_{}^{}{}_{}^{}}\end{bmatrix} + p_{0}}$ or approximated by $p = {{c\begin{bmatrix}{\sin^{- 1}{\,^{u}x}\square_{y}} \\{\sin^{- 1}{\,^{u}x}\square_{z}}\end{bmatrix}} + \begin{bmatrix}{{}_{}^{}{}_{}^{}} \\{{}_{}^{}{}_{}^{}}\end{bmatrix} + p_{0}}$ where: q is said angular position; ^(u)u_(y) isa detected value of the linear position y-axis in a user's frame ofreference; ^(u)u_(z) is a detected value of the linear position x-axisin a user's frame of reference; ^(u)x°_(x) is a detected value of thebody-frame x-axis in a user's frame of reference; ^(u)x°_(y) is adetected value of the body-frame y-axis in a user's frame of reference;^(u)x°_(z) is a detected value of the body-frame z-axis in a user'sframe of reference; and p₀ is the 2D vector value of the displaycoordinate system origin in cursor coordinates.
 9. The method of claim1, wherein at least one of the mapping algorithms creates anintermediate, virtual display that moves to face the device.
 10. Themethod of claim 9, wherein said wherein said intermediate, planarvirtual display maps device motion to cursor location based upon anintersection of a forward pointing direction (body-frame x-axis) of thedevice and a surface of a display on which said cursor is to bedisplayed rotated to at least partially face the device.
 11. The methodof claim 10, wherein said intermediate, planar virtual display isdefined as: $Q = \begin{bmatrix}| & | & | \\{{}_{}^{}{}_{}^{}} & {{}_{}^{}{}_{}^{}} & {{}_{}^{}{}_{}^{}} \\| & | & |\end{bmatrix}^{T}$ s = Q ^(u)x▫w = Q ^(u)u$p = {{\frac{w_{x}}{s_{x}}\begin{bmatrix}s_{y} \\s_{z}\end{bmatrix}} + \begin{bmatrix}w_{y} \\w_{z}\end{bmatrix} + p_{0}}$ where: ^(u)x^(Δ) is a detected value of thevirtual display x-axis in a user-frame of reference; ^(u)y^(Δ) is adetected value of the virtual display y-axis in a user-frame ofreference; ^(u)z^(Δ) is a detected value of the virtual display z-axisin a user-frame of reference; w_(y) is a detected value of the linearposition y-axis in the virtual display frame of reference; w_(z) is adetected value of the linear position z-axis in the virtual displayframe of reference; s_(x) is a detected value of the body-frame x-axisin the virtual display frame of reference; s_(y) is a detected value ofthe body-frame y-axis in the virtual display frame of reference; s_(z)is a detected value of the body-frame z-axis in the virtual displayframe of reference; and p₀ is the 2D vector value of the displaycoordinate system origin in cursor coordinates.
 12. The method of claim1, wherein at least one of the mapping algorithms creates anintermediate, spherical virtual display that moves to face the device.13. The method of claim 12, wherein said spherical virtual display mapsdevice motion to cursor location based upon spherical coordinates of theestimated angular position being transformed into the virtual displaycoordinates by a transformation matrix and converted into cursorlocation.
 14. The method of claim 13, wherein said mapping is performedwithout using said estimated linear position of said device.
 15. Themethod of claim 14, wherein said spherical virtual display is definedas: φ = sin⁻¹(−b_(z))$\theta = {{{\tan^{- 1}\left( \frac{b_{y}}{b_{x}} \right)}\begin{bmatrix}p_{x} \\p_{y} \\1\end{bmatrix}} = {T\begin{bmatrix}\theta \\{- \varphi} \\1\end{bmatrix}}}$ where: b_(x) is the detected value of the body-framex-axis in the virtual display frame of reference; b_(y) is the detectedvalue of the body-frame y-axis in the virtual display frame ofreference; b_(z) is the detected value of the body-frame z-axis in thevirtual display frame of reference; and T is a 3×3 transformation matrixof arbitrary constants which may apply any combination of a scale,rotation, translation, shearing, reflection, orthogonal projection,affine transformation and a perspective projection.
 16. The method ofclaim 1, wherein at least one of the mapping algorithms creates anintermediate, relative, spherical virtual display which uses a change inangle between the device and the display to determine a change in cursorlocation.
 17. The method of claim 16, said intermediate, relative,spherical virtual display maps device motion to cursor location basedupon body-frame angular velocity modified by a transformation matrix andconverted into cursor change in location. 18-28. (canceled)
 29. A 3Dpointing device comprising: at least one sensor configured to generatean output which is associated with movement of said 3D pointing device;and a processor configured to estimate at least one of said device'slinear position and angular position using said output, to process atleast one of said estimated linear position and angular position usingboth a first mapping algorithm to generate a first cursor location and asecond mapping algorithm to generate a second cursor location, and tocombine said first cursor location and said second cursor location togenerate a final cursor output.
 30. The 3D pointing device of claim 29,wherein said first mapping algorithm is an absolute pointing algorithmand wherein said second mapping algorithm is a relative pointingalgorithm.
 31. The 3D pointing device of claim 29, wherein at least oneof the mapping algorithms is an absolute pointing algorithm which has acharacteristic that a ratio of cursor motion to angular device motionchanges as a function of distance between said device and a display onwhich a cursor is displayed.
 32. The 3D pointing device of claim 31,wherein said absolute pointing algorithm maps device motion to cursorlocation based upon an intersection of a forward pointing direction(body-frame x-axis) of the device and a surface of a display on whichsaid cursor is to be displayed.
 33. The 3D pointing device of claim 32,wherein said absolute pointing algorithm is defined as:${p = {{- {\frac{{}_{}^{}{}_{}^{}}{{\,^{u}x}\square_{x}}\begin{bmatrix}{{\,^{u}x}\square_{y}} \\{{\,^{u}x}\square_{z}}\end{bmatrix}}} + \begin{bmatrix}{{}_{}^{}{}_{}^{}} \\{{}_{}^{}{}_{}^{}}\end{bmatrix} + p_{0}}},$ where ^(u)u_(x) is a detected value of thelinear position x-axis in a user's frame of reference; ^(u)u_(y) is adetected value of the linear position y-axis in a user's frame ofreference; ^(u)u_(z) is a detected value of the linear position z-axisin a user's frame of reference; ^(u)x°_(x) is a detected value of thebody-frame x-axis in a user's frame of reference; ^(u)x°_(y) is adetected value of the body-frame y-axis along an y-axis in a user'sframe of reference; ^(u)x°_(z) is a detected value of the body-framez-axis along a z-axis in a user's frame of reference; and p₀ is the 2Dvector value of the display coordinate system origin in cursorcoordinates.
 34. The 3D pointing device of claim 29, wherein at leastone of the mapping algorithms is an absolute invariant algorithm whichhas a characteristic of providing a direct, repeatable mapping fromdevice linear position and angular position into cursor location and asecond characteristic that cursor responsiveness to linear motion andangular motion is consistent over a motion range. 35-106. (canceled)